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Abstract 


This  technical  note  describes  the  synergistic  application  of  the  balanced  scorecard  and  goal- 
driven  measurement  methodologies  to  develop  measures  and  associated  indicators  for 
measuring  an  organization’s  health  and  performance.  Through  this  iterative  approach,  an 
organization’s  strategic  goals  and  subgoals  are  mapped  to  the  balanced  scorecard  and  refined. 
The  goal-question-(indicator)-measurement  methodology  is  then  applied  to  identify 
indicators  and  measures  for  each  scorecard  dimension.  A  hypothetical  example  of  how  to 
apply  the  methodology  at  a  “typical”  organization  performing  software  development  and 
maintenance  activities  is  provided.  The  example  yields  typical  indicators  to  illustrate  the 
methodology. 
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Introduction 


Many  organizations,  including  government  agencies,  define  enterprise-wide  measures  to 
reflect  the  relative  health  of  their  organization.  These  measures  help  guide  an  organization’s 
overall  performance  and  process  improvement  effort.  Two  methodologies  often  employed  to 
develop  enterprise-wide  measures  are  the  balanced  scorecard  [Kaplan  01]  and  goal-driven 
measurement1  [Park  96]  methodologies. 

Both  methodologies  are  well  known,  but  usually  applied  separately.  This  technical  note 
suggests  an  approach  for  combining  the  techniques,  taking  advantage  of  the  best  of  each.  The 
balanced  scorecard  encourages  an  organization  to  take  an  introspective  look  at  its  practices. 
From  this,  the  organization  can  set  enterprise-strategic  goals  and  develop  a  set  of  indicators 
and  measurements  for  the  desired  outcomes  and  performance  drivers.  The  goal-question- 
(indicator)-measurement  (GQ[I]M)  approach  then  follows  as  a  disciplined  way  for  deriving 
the  required  measures  and  indicators.  The  approach  is  intended  to  help  an  organization 
determine  the  best  measures  and  associated  indicators  for  its  unique  environment.  Using  this 
approach,  an  organization  can  systematically  set  goals  for  each  of  the  perspectives  of  the 
balanced  scorecard  and  develop  a  set  of  strategic  measures  and  indicators  to  determine  and 
track  the  quality  of  outcomes  and  organizational  performance. 

This  technical  note  begins  with  a  description  of  the  approach  to  combine  the  goal-driven 
measurement  and  balanced  scorecard  methodologies.  Next,  as  an  example,  results  from 
applying  the  approach  to  a  “typical”  software  development  and  maintenance  organization  are 
provided,  with  the  recommended  measures  and  indicators  that  were  defined  to  reflect  that 
organization’s  health  and  performance.  We  note  that  these  measures  and  indicators  are 
examples  only.  Each  organization  must  develop  its  own  set  of  measures  and  indicators  in  the 
context  of  its  business  goals,  which  is  precisely  what  the  approach  is  designed  to  accomplish. 
We  defer  discussions  on  techniques  to  analyze  resultant  indicators  and  to  develop  plans  to 
achieve  the  organization’s  business  goals  for  subsequent  technical  notes. 


1.1  The  Balanced  Scorecard  Framework 

The  balanced  scorecard  is  an  industry-recognized  best  practice  for  measuring  the  health  of 
an  organization.  It  can  be  used  as  a  management  tool  for  translating  an  organization’s  mission 
and  strategic  goals  into  a  comprehensive  set  of  performance  measures  that  provide  the 
framework  for  an  enterprise  measurement  and  management  system  [Castro  02]. 

1  We  use  the  word  “measurement”  instead  of  “metric”  throughout  this  report.  Measurement  is 
defined  as  1)  a  standard  or  unit  of  measurement;  2)  a  quantitative  indication  of  the  extent,  amount, 
dimension,  capacity,  or  size  of  some  attribute  of  a  product  or  process. 
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The  balanced  scorecard  methodology  is  based  on  four  perspectives  of  an  organization’s 
performance — customer,  financial,  internal  process,  and  learning  and  growth.  These 
perspectives  are  illustrated  in  Figure  1. 


Figure  1:  The  Balanced  Scorecard 


Using  the  balanced  scorecard  framework,  an  organization  can  systematically  set  enterprise 
strategic  goals  for  each  perspective  and  develop  a  set  of  indicators  and  measurements  for  the 
desired  outcomes  and  performance  drivers  that  will  enable  the  achievement  of  the  enterprise 
outcomes.  The  result  is  a  set  of  interconnected  goals  and  measurements  with  defined  cause- 
and-effect  relationships. 

As  a  template,  the  balanced  scorecard  can  be  applied  to  most  businesses.  It  ensures  that  a 
resultant  set  of  measures  will  provide  coverage  of  the  basic  elements  of  organizational 
performance.  Different  market  situations,  product  strategies,  motivations,  and  competitive 
environments  require  different  scorecards.  Therefore,  a  scorecard  should  be  developed  to  fit 
the  organization  and  its  mission.  Every  organization  is  unique  and  follows  its  own  path  to 
develop  a  balanced  scorecard.2 

Companies  including  Mobil  Oil  Corporation’s  North  American  Marketing  and  Refining 
Division,  CIGNA  Corporation’s  Property  &  Casualty  Division,  Chemical  Retail  Bank  and 
Brown  &  Root  Energy  Services’  Rockwater  Division,  Xerox,  and  Motorola  have  used  the 
balanced  scorecard  approach  [Kaplan  01].  Federal  agencies,  such  as  the  U.S.  Department  of 
Energy,  use  the  Balanced  Scorecard  for  Government  Procurement. 


2  To  better  reflect  how  the  balanced  scorecard  was  used  in  this  work,  we  have  added  “mission”  as 
part  of  the  central  driving  point  and  changed  “Learning  and  Growth”  to  “Innovation  and  Learning” 
as  a  more  descriptive  heading.  Figure  1  is  derived  from  a  balanced  scorecard  figure  presented  in  “A 
Management  Guide  for  the  Deployment  of  Strategic  Metrics”  developed  at  Raytheon  Corporation. 
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1 .2  GQ(I)M  Methodology 

The  goal-driven  measurement  process  aligns  measures  and  indicators  with  goals,  thus 
ensuring  that  the  measures  and  indicators  selected  will  be  used  to  show  success  in  achieving 
these  goals. 

In  our  elaboration  of  Basili’s  process  [Park  96],  we  have  added  an  intermediate  step  to  assist 
in  linking  the  questions  to  the  measurement  data  that  will  be  collected.  The  importance  of 
linking  data  to  the  questions  they  answer  is  clear  in  the  success  Basili  has  had  with  the  GQM 
approach.  Our  experience  suggests  that  identifying  questions  and  measures  without 
visualizing  an  indicator  is  often  not  sufficient  to  get  a  successful  measurement  program 
started.  The  displays  or  reports  used  to  communicate  the  data  (called  indicators  in  our 
variation  of  the  GQM  methodology)  are  a  key  link  that  can  determine  the  success  or  failure 
of  a  measurement  program.  These  indicators  serve  as  a  requirements  specification  for  the 
data  that  must  be  gathered,  the  processing  and  analysis  of  the  measurement  data  that  must 
take  place,  and  the  schedule  by  which  these  activities  occur. 

In  the  methodology,  business  or  enterprise  strategic  goals  are  translated  into  measurement 
goals  [Basili  84],  [Briand  96]  by  first  identifying  high-level  business  goals  and  then  refining 
them  into  concrete  operational  statements  or  subgoals  with  a  measurement  focus.  This 
refinement  process  involves  probing  and  expanding  each  high-level  goal  by  using  it  to  derive 
quantifiable  questions  whose  answers  would  assist  managing  the  organization.  The  questions 
provide  concrete  examples  that  can  lead  to  statements  that  identify  the  type  of  information 
needed.  From  these  questions,  displays  or  indicators  are  postulated  that  provide  answers  and 
help  link  the  measurement  data  that  will  be  collected  to  the  measurement  goals.  The  displays 
or  reports  used  to  communicate  the  data  are  a  key  link  to  understanding  just  why  the  specific 
data  are  being  collected.  The  overall  process  is  illustrated  in  Figure  2.  The  figure  assumes 
that  vision  and  mission  statements  exist. 
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Business  =>  Sub  goals  =>  Measurement 
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What  do  i  want  to  know  or  learn? 
How  do  I  know  if  I  have  met  my  goal?| 
(Success  Criteria) _ 


SLOC  Staffhours  Trouble  Reports  Milestone  dates 
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j 

V 

V 

Indicator  Template 

Objective 
Question 


rm 


Inputs 
Algorithm 


Assumptions: 


Infrastructure 
I  Assessment 


Figure  2:  Goal-Driven  (Indicator)  Measurement  Process 


The  goal-driven  measurement  methodology  as  implemented  at  the  Software  Engineering 
Institute  (SEIsm)  is  presented  in  Appendix  A. 


Note  that  the  organization’s  strategic  goals  are  used  in  both  the  GQ(I)M  and  balanced 
scorecard  methodologies.  In  this  way,  the  two  methodologies  seem  redundant.  However,  the 
use  of  the  balanced  scorecard  forces  an  introspection  of  the  organization  against  a  standard 
way  of  examining  the  health  of  an  organization  and  organizing  the  purpose  of  the  eventual 
measures  and  indicators.  The  GQ(I)M  then  follows  as  a  disciplined  way  to  derive  the  specific 
measures  and  indicators  needed. 


SM  SEI  is  a  service  mark  of  Carnegie  Mellon  University. 
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2  Suggested  Approach 


Figure  3  illustrates  our  suggested  process/approach  for  developing  enterprise-wide  measures 
and  associated  indicators  that  reflect  an  organization’s  health  and  performance.  The  process  is 
typically  iterative  and  contains  the  following  four  steps: 

1.  obtain  and  clarify  mission  and  vision  statements 

2.  derive  strategic  goals  and  subgoals  using  GQ(I)M 

3.  map  subgoals  to  balanced  scorecard 

4.  apply  GQ(I)M  to: 

a.  define  success  criteria  for  each  subgoal 

b.  pose  relevant  questions  and  postulate  indicators  that  address  each  subgoal  in  each 
quadrant  of  the  BSC 

c.  determine  requisite  measures  or  data  elements  that  allow  indicators  to  be  crafted 
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Figure  3:  Overview  of  the  Approach 


2.1  Obtain  and  Clarify  Mission  and  Vision  Statements 

The  mission  statement  describes  the  organization’s  statement  of  purpose;  what  it  is  doing 
and  why.  The  vision  statement  describes  what  the  organization  wants  to  aspire  to,  the 
organization’s  special  task,  and  what  specific  motivation  binds  together  the  organization’s 
stakeholders — including  members,  leaders  and  anyone  else  affected  by  the  issue.  Vision 
reflects  the  realization  of  the  organization’s  values.  The  understanding  of  values  provides 
motivation  and  community.  Appendix  B  contains  more  detailed  advice  for  generating  mission 
and  vision  statements. 
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Advice: 


•  Use  an  iterative  approach  to  create  clear  and  meaningful  mission  and  vision  statements 
that  describe  the  organization’s  unique  purpose.  The  approach  described  here  provides 
for  this  iterative  technique. 

•  Avoid  immediate  “wordsmithing.”  Work  out  the  basic  concepts  first,  then  refine. 


2.2  Apply  GQ(I)M  to  Derive  Strategic  Goals  and  Subgoals 

Once  an  organization  understands  its  mission  and  vision  and  sets  its  sights  on  what  it  wants 
to  aspire  to,  it  can  develop  its  strategic  goals.  Strategic  goals  are  analogous  to  operational 
requirements,  from  which  an  organization  can  derive  more  detailed  goals  and  activities  for 
achieving  its  vision.  Strategic  goals  lay  the  foundation  for  this  planning  and  help  outline  the 
approach.  One  example  of  a  strategic  goal  is  the  statement:  “Increase  customer  satisfaction  to 
get  a  bigger  market  share.”  Here,  the  goal  has  quantitative  expression  of  achievement — “to 
get  a  bigger  market  share.”  In  this  example,  as  is  typically  the  case,  there  are  dependencies 
within  the  goal  statement  itself.  That  is,  one  goal  statement  combines  concepts  from  more 
than  one  goal.  Here,  the  goal  is  to  increase  customer  satisfaction.  But  the  second  part  is  also  a 
goal  “to  get  a  bigger  market  share.”  As  stated,  these  seem  like  two  distinct  goals. 


Advice:  Use  structured  brainstorming  or  the  nominal  group  technique  (NGT)  to  generate 
a  list  of  the  strategic  goals  [Scholtes  90].  Merge  similar  goals  and  sort  the  results  into  a 
prioritized  list. 


Because  the  strategic  goals  are  purposely  at  a  high  level  of  abstraction,  it  is  necessary  to 
derive  subgoals  using  the  GQ(I)M  methodology  in  combination  with  the  vision,  mission,  and 
strategic  goals.  Subgoals  are  analogous  to  functional  requirements,  which  an  organization 
must  plan  and  implement  to  satisfy  not  only  its  subgoals  but  its  strategic  goals  as  well.  For 
example,  “Increase  market  share  by  15%  in  the  next  fiscal  year”  is  a  subgoal. 

For  the  most  part,  when  subgoals  are  articulated  with  a  quantitative  component,  and  possibly 
an  element  of  timing,  this  provides  the  requisite  picture  of  success.  It  is  important  to  realize 
that  there  may  be  many  subgoals.  It  is  beneficial  for  the  organization  to  prioritize  them  to 
develop  a  meaningful  and  efficient  strategy  for  achieving  them,  rather  than  trying  to  achieve 
every  subgoal. 


CMU/SEI-2003-TN-024 


Advice:  Goals  are  often  a  function  of  where  you  sit  [Lynch  91].  Since  the  goals  in 
hierarchical  organizations  are  related,  ensure  traceability  of  subgoals  back  to  the  primary 
strategic  goal.  The  interpretation  of  the  goal  will  differ  at  different  places  in  the 
organization.  It  is  important  to  understand  these  different  perspectives  and  trace  the 
subgoals  back  to  the  primary  strategic  goal. 


2.3  Map  Subgoals  to  Balanced  Scorecard 

In  this  step,  the  subgoals  are  “mapped”  or  allocated  into  the  quadrants  of  the  balanced 
scorecard.  The  four  quadrants  are  inherently  linked  together.  In  allocating  the  subgoals,  some 
quadrants  may  have  some  of  the  same  subgoals  assigned  to  them.  This  mapping  is  iterative 
since  the  initial  mission,  strategic  goal  statements,  and  subgoal  statements  may  not  seem  to  fit 
the  balanced  scorecard.  However,  in  achieving  this  step,  an  organization  can  determine  if  its 
mission,  goals,  and  subgoals  are  stated  correctly.  This  iterative  potential  is  crucial  for 
developing  effective  measures  and  indicators  that  reflect  the  health  of  the  organization  and 
are  clearly  linked  to  its  strategic  goals. 

In  performing  this  mapping  we  asked  questions  appropriate  to  each  scorecard  quadrant. 

From  a  Customer  Perspective,  how  do  customers  see  the  organization?  Typically,  concerns 
fall  into  four  categories:  time,  quality,  performance,  and  service.  But  each  of  these  categories 
must  be  defined  or  translated  into  the  organization’s  terminology  and  the  way  it  conducts  its 
business.  For  example,  time  may  mean  the  response  of  the  organization  to  the  customer  needs 
or  the  time  it  takes  to  process  a  customer  order  and  deliver  a  product.  Then  we  asked  what 
subgoals  contribute  directly  or  indirectly  to  these  categories. 

From  an  Internal  Business  Perspective,  what  must  we  excel  at  to  support  our  customer 
focus?  That  is,  which  internal  strategies  or  operations  have  the  greatest  impact  on  customer 
satisfaction?  Typically,  the  factors  considered  are  cycle  time,  quality  of  work  products  or 
services,  employee  skills,  and  productivity.  These  categories  must  be  defined  or  translated 
into  the  organization’s  terminology  and  the  way  it  conducts  its  business.  As  before,  we  can 
ask  which  subgoals  have  we  defined  that  contribute  directly  or  indirectly  to  these  categories. 

For  the  quadrant  of  Innovation  and  Learning,  are  we  interested  in  our  ability  to  improve 
and  create  value?  Typical  concerns  include:  How  much  is  invested  in  Innovation  and 
Learning?  Does  it  take  more  or  less  time  for  an  employee  to  achieve  a  level  of  fluency  in  his 
job?  How  many  new  products  have  we  introduced?  What  increase  in  revenue  is  due  to  new 
products?  This  focuses  on  an  organization’s  ability  to  innovate,  improve,  and  leam.  It  is 
directly  tied  to  the  organization’s  value,  in  both  product  and  process  innovation. 
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In  the  Financial  Perspective  quadrant,  the  question  is  how  do  we  look  to  shareholders?  (For 
many  organizations,  where  profitability  is  defined  as  revenue,  shareholders  may  be  replaced 
by  stakeholders.)  Here  we  look  at  profitability,  growth,  and  value  to  stakeholders. 

Table  1  presents  an  additional  set  of  questions  for  each  quadrant  of  the  balanced  scorecard. 
Answers  to  these  questions  provide  further  insight  into  the  appropriate  mapping  of  subgoals 
and,  ultimately,  the  needed  measures  and  indicators.  These  questions  should  be  generated  by 
the  organization. 
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Table  1:  Questions  for  Each  Quadrant  of  the  Balanced  Scorecard 


Balanced  Scorecard  Quadrant 

Questions 

Customer 

What  is  important  to  our  customers?  What  are  their  “hot  buttons?” 

How  do  customers  evaluate  our  timeliness? 

What  do  customers  consider  a  quality  product?  Are  there  any  standards  or 
goals  currently  set  by  our  customers? 

How  and  what  do  customers  currently  evaluate  our  organization? 

In  the  financial  area,  what  is  important  to  our  customers? 

What  does  responsive  mean  to  our  customers?  Are  any  goals  or  standards 
set? 

Internal  Business 

How  do  we  define  quality — a  product  with  few  deficiencies?  How  do  we 
define  success — a  quality  product?  How  we  know  we’re  successful? 

How  do  we  define  the  completion  of  a  trouble  report?  Are  any  goals  set 
on  the  length  of  time  it  should  take  to  close/fix  trouble  reports? 

How  are  process  improvement  requirements  determined  and  how  is  their 
impact  currently  measured? 

How  is  productivity  measured?  Is  it  important?  Are  goals  or  targets  set? 

How  do  we  define  and  measure  “cycle  time?”  Are  improvement  goals  or 
targets  set? 

How  do  you  do,  and  who  does,  the  Quality/Schedule/Cost  tradeoffs? 

Innovation  &  Learning 

What  attributes  of  quality  are  currently  measured?  Are  any  goals  set? 

Are  there  any  ongoing  process  improvement  activities?  Any 
measurements  being  taken  as  to  their  impact?  Are  any  targets  or  goals 
set? 

Is  employee  satisfaction  currently  measured? 

What  activities  are  being  modified  to  improve  responsiveness? 

Financial 

How  do  you  know  if  the  organization  has  effective  financial  control? 

What  defines  success?  How  do  we  know  if  we  are  successful? 

How  is  the  cost/performance  tradeoff  accomplished? 

Are  there  any  ongoing  improvement  activities?  Are  any  targets  or  goals 
set? 

Advice:  Use  the  iterative  potential  of  mapping  subgoals  to  the  balanced  scorecard  to  assist  in 
determining  if  the  mission,  goals,  and  subgoals  are  stated  correctly. 
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2.4  Specify  Success  Criteria 

So  far  in  the  methodology,  we  have  clarified  the  mission  and  vision  statements,  derived  our 
strategic  goals,  further  decomposed  them  into  more  definitive  subgoals,  and  mapped  them  to 
the  quadrants  of  the  balanced  scorecard.  As  part  of  the  mapping  step  we  need  to  articulate  the 
success  criteria  for  the  subgoals.  To  aid  in  specifying  the  success  criteria,  it  is  very  useful  to 
visualize  what  success  looks  like  before  articulating  the  actual  success  criteria  for  each 
subgoal  of  the  balanced  scorecard.  For  the  most  part,  if  the  subgoals  are  articulated  with  a 
quantitative  component,  and  possibly  an  element  of  timing,  this  would  provide  the  requisite 
picture  of  success. 

For  each  subgoal,  the  following  questions  should  be  addressed: 

•  How  do  you  define  success? 

•  How  do  you  know  when  you  have  reached  success? 

•  What  are  the  attributes  of  success? 

Table  1  can  also  be  used  as  a  basis  for  formulating  the  success  criteria. 


Advice:  Articulate  clearly  the  criteria  you  will  use  to  decide  if  the  goal  has  been  met. 


2.5  Apply  GQ(I)M  to  Derive  Measures  and  Indicators 

This  step  is  subdivided  into  a  number  of  smaller  steps,  which  are  described  below.  Again, 
these  steps  can  be  performed  iteratively. 

1.  Pose  relevant  questions  and  postulate  indicators  that  address  each  success  criterion 
for  each  subgoal  in  each  quadrant  of  the  balanced  scorecard. 

When  identifying  questions  and  defining  indicators,  it  is  important  to  keep  in  mind  the 
goal(s)  you  are  addressing  and  how  the  measurement  results  will  be  used.  For  each  subgoal, 
develop  questions  related  to  the  subgoal  and  that  would  assist  in  determining  if  the  success 
criteria  have  been  met.  Phrase  these  questions  in  a  manner  that  elicits  a  quantitative  response. 
Using  the  answers  to  these  questions,  some  general  indicators  can  be  described. 

The  specific  indicators  that  should  be  used  in  each  of  the  four  balanced  scorecard  dimensions 
depend  upon  the  specific  questions  or  issues  addressed.  Once  the  questions  or  issues  are 
known,  a  specific  indicator  can  be  designed.  It  is  recommended  that  one  indicator  be 
developed  for  each  question.  After  doing  this,  it  may  be  possible  to  combine  indicators.  This 
should  only  be  done  after  careful  analysis  of  the  individual  indicators.  The  indicators  in  this 
document  are  examples  of  some  indicators  that  have  been  found  useful  in  other  organizations 
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with  similar  questions  and  concerns.  They  are  only  examples  to  illustrate  the  methodology 
and  should  not  be  adopted  as-is  by  organizations. 

See  Appendix  C  for  additional  information  about  using  indicators. 

2.  Determine  requisite  measures  or  data  elements  that  allow  indicators  to  be  crafted. 

In  this  substep,  the  data  elements  or  measures  required  to  construct  the  indicators  are 
identified. 

3.  Document  results. 

Consistent  with  the  GQ(I)M  methodology  [Park  96],  indicator  templates  are  used  to 
document  items  such  as  why,  what,  who,  where,  when,  and  how  for  each  indicator.3  The 
completed  templates  are  collected  in  a  measurement  or  indicator  handbook.  The  template 
includes  fields  for 

•  precise  objective  of  the  indicator 

•  visual  display 

•  inputs 

•  algorithms 

•  assumptions 

•  data  collection  information 

•  data  reporting  information 

•  analysis  and  interpretation  of  results 

This  template  is  part  of  the  SEI  methodology  described  in  Appendix  D.  Organizations  tend  to 
tailor  the  template  to  fit  their  environment  by  adding,  modifying,  or  deleting  fields  in  advance 
of  specifying  a  set  of  indicators. 


Many  organizations  have  recognized  the  importance  of  using  precise  communication  and 
collecting  measurements  based  on  need  rather  than  capability  [Augustine  99].  While  the  steps 
described  in  this  approach  seem  daunting,  they  provide  a  structured  way  for  ensuring  that  an 
organization  uses  the  balanced  scorecard  and  GQ(I)M  methodologies  correctly  and  to  their 
full  potential. 


Advice:  Follow  the  steps  in  the  described  methodology  to  better  understand  to  formulate 
meaningful  measures. 


3  This  information  is  extracted  from  the  Implementing  Goal-Driven  Software  Measurement  course 
offered  by  the  SEI. 
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3  Example  Approach 


This  section  describes  how  the  above  approach  can  be  applied  to  a  “typical”  organization. 
Here,  typical  is  defined  as  an  aggregate  of  several  organizations  with  similar 
characteristics.  In  this  example,  the  organization  is  a  government  agency  consisting  of 
300  management,  administrative,  and  technical  personnel.  (Please  note  that  the  overall 
approach  is  not  dependent  on  any  single  organization  type.)  The  indicators  presented  here 
are  just  examples  to  illustrate  the  methodology.  They  should  not  be  adopted  as-is  by 
organizations. 


3.1  Obtain  and  Clarify  Mission  and  Vision  Statements 

The  stated  mission  after  clarification  is: 

Develop  and  maintain  information  technologies  and  integrated  software-intensive 
<xyz>  systems  for  die  government. 

Here,  development  is  the  “in-house”  development  of  software  and  systems  to  be  fielded; 
maintenance  is  the  maintenance  and  enhancement  of  the  system  components  of  these 
fielded  systems.  <xyz>  is  a  domain  such  as  Command  and  Control. 

The  stated  vision  after  clarification  is: 

We  are  recognized  as  the  premier  government  organization  in  the  development 
and  maintenance  of  superior  information  technologies  and  integrated  software¬ 
intensive  <xyz>  systems. 

3.2  Apply  GQ(I)M  to  Derive  Strategic  Goals  and  Subgoals 

Using  the  mission  and  vision  statements,  the  organization’s  strategic  goals  as  initially 
stipulated  by  senior  management  are 

•  provide  the  best  value  system/software  engineering  environment  and  processes 

•  improve  the  organization’s  financial  posture 

•  provide  on  time  delivery  of  high-quality  systems 

•  provide  leadership  for  the  government  in  technology  development  and  insertion  to 
accomplish  the  mission 

•  provide  a  world-class  quality  of  life  for  employees 
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These  are  qualitative  views  of  goals  that  senior  management  believes  will  lead  to  a 
strategy  for  improving  the  organization’s  performance.  These  must  be  further  refined  to 
eventually  provide  the  organization,  at  various  levels,  targets  for  fulfilling  its  vision. 
Following  the  suggested  approach  in  applying  the  GQ(I)M  methodology,  the  subgoals  are 
derived  from  the  strategic  goals,  vision,  and  mission.  Note  the  quantitative  aspect 
embedded  in  some  of  these  subgoals: 

•  provide  the  best  value  system/software  engineering  environment 

-  improve  processes.  Improve  the  quality  of  internal  processes  and  work  products  by 
achieving  Capability  Maturity  Model®  Integration  (CMMI®)  maturity  level  5  in  the 
next  year. 

-  improve  communications  (internal).  Improve  communication  mechanisms  within  the 
organization  and  with  external  stakeholders. 

-  minimize  rework.  Reduce  the  mean  of  the  distribution  of  deficiencies  detected  per 
development  life-cycle  phase  by  15%. 

•  improve  the  organization’s  financial  posture 

-  effective  financial  controls.  Develop  and  improve  mechanisms  to  provide  visibility 
into  the  organization’s  financial  situation  on  a  weekly  basis. 

-  funding  stability.  Develop  methods  to  mitigate  and  accommodate  funding  variations 
of  15%. 

-  delivered  costs.  Decrease  costs  of  delivered  products  and  services  by  10%. 

-  increased  mission  funding.  Increase  mission  funding  by  10%. 

-  customer  funding.  Increase  funding  from  customers  by  10%  over  the  next  year  and 
expand  the  customer  base  to  at  least  two  private  corporations. 

•  provide  on-time  delivery  of  high-quality  systems 

-  quality  of  delivered  products.  Quality  deficiencies  detected  in  the  delivered  product 
over  one  year  period  decrease  by  10%.  Quality  is  defined  as  increased  compliance 
with  customer  requirements  and  increased  on-time  delivery  (less  than  10%  late 
delivery). 

-  timeliness.  Reduce  the  time  needed  to  transition  from  concept  to  operational 
capability  by  10%  in  the  next  delivery  over  the  current  expected  time  (reduced  cycle 
time). 

-  communications  (external).  Improve  communication  mechanisms  with  customers  to 
ensure  that  responses  to  requests  are  made  within  two  working  days. 

-  resources.  Increase  availability  and  capability  of  internal  resources  to  ensure  that 
customer  needs  are  satisfied  90%  of  the  time. 

-  relationships.  Understand  and  anticipate  our  customers’  needs  and  exceed  their 
expectations. 

•  provide  leadership  for  the  government  in  technology  development  and  insertion  to 
accomplish  the  mission 

-  improve  business.  Become  an  integral  part  of  government  transformation  and 
homeland  defense  this  fiscal  year. 

-  process  improvement.  Achieve  CMMI  maturity  level  5. 


®  Capability  Maturity  Model  and  CMMI  are  registered  in  the  U.S.  Patent  and  Trademark  Office  by 
Carnegie  Mellon  University. 
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-  new  technologies.  Develop  and  implement  technologies  that  provide  the  capability  to 
increase  efficiency  of  operations  by  5%. 

•  provide  a  world-class  quality  of  life  for  employees 

-  process  improvement.  Improve  the  quality  of  internal  processes  and  work  products 
by  achieving  CMMI  maturity  level  5  in  the  next  year. 

-  enhance  staff  capability.  Recruit,  develop,  and  retain  a  well-trained,  competent, 
diverse,  value-based  workforce  in  an  environment  that  fosters  innovation  and  is 
intolerant  of  discrimination. 

-  improve  communications  (internal).  Improve  communication  mechanisms  within  the 
organization  and  with  external  stakeholders. 

-  quality  of  life  (employee  satisfaction).  Provide  upwardly  mobile  careers  for 
employees  and  lower  the  turnover  rate  by  15%  in  the  next  year. 

-  new  technologies.  Develop  and  implement  technologies  that  provide  the  capability  to 
increase  efficiency  of  operations  by  5%. 

3.3  Map  Subgoals  to  Balanced  Scorecard 

The  following  figure  shows  the  results  of  mapping  the  organization’s  subgoals  to  each 
quadrant  of  the  balanced  scorecard  where  indicators  need  to  be  postulated.  Only  the  titles  of 
the  subgoals  are  shown  for  clarity. 


r 


Customer 

Quality  of  Delivered  Products 
Timeliness 

Communications  (external) 

Funding 

Resources 

Relationship 


% 


Financial 

i 

£ 

k 

Effective  financial  controls 
Funding  stability 

Delivered  costs 

Increase  mission  funding 

A 

Mission 

HHHkV 

Resources 

Increase  mission  funding 

, 

KH 

Engineering  posture 

Minimize  rework 

Improve  communications 
(internal) 

N 

vision 

Customer  Funding 

P 

3 

J 

i 

▲ 

Innovation  and  Learning 

Process  improvement 
Enhance  staff  capability 
Improve  communications  (internal) 
Quality  of  life  (employee  satisfaction)] 
New  technologies 


Figure  4:  Mapping  Subgoals  to  the  Balanced  Scorecard 
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As  part  of  this  mapping,  we  are  also  required  to  define  success  criteria.  Here,  we  reexamine 
the  subgoals  to  see  if  they  address  the  following  questions: 

•  How  do  you  define  success? 

•  How  do  you  know  when  you  have  reached  success? 

•  What  are  the  attributes  of  success? 

Generally,  if  the  subgoals  are  articulated  with  a  quantitative  component,  and  possibly  an 
element  of  timing,  this  would  provide  the  requisite  picture  of  success.  The  subgoals  derived 
for  this  example  have  the  needed  quantitative  and  timing  aspects  to  answer  these  questions. 
In  this  example,  please  note  that  every  subgoal  does  not  map  to  the  balanced  scorecard.  This 
only  indicates  that  a  particular  subgoal  might  not  be  appropriate  or  might  need  to  be 
reconsidered. 


3.4  Pose  Relevant  Questions  and  Postulate  Indicators 

With  the  mapping  to  the  balanced  scorecard  to  ensure  the  full  spectrum  of  the  organization’s 
business  posture  is  addressed,  the  GQ(I)M  method  is  applied  to  develop  relevant  questions 
and  postulate  indicators.  This  section  presents  indicators  for  the  customer  quadrant  of  the 
balanced  scorecard.  Quantifiable  questions  are  developed  for  each  of  the  subgoals.  Indicators 
in  the  form  of  tables,  charts,  graphs,  etc.  that  address  the  questions  are  postulated. 

The  following  subgoals  will  be  addressed  to  illustrate  the  methodology  for  the  customer 
quadrant  of  the  balanced  scorecard: 

•  quality  of  delivered  products 

•  timeliness 

•  resources 

3.4.1  Subgoal:  Quality  of  Delivered  Products 


Customer 

-  Quality  of  delivered 
products 

-  Timeliness 

-  Resources 

|  Financial  | 

A  Balanced 
Perspective 

i  Internal 

|  Business 

Innovation 
k  and  Learning 

IX 

The  subgoal  of  quality  of  delivered  products  for  the  customer  quadrant  is  shown  below. 
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Subgoal:  quality  of  delivered  products.  Quality  deficiencies  detected  in  the  delivered 
product  over  a  one-year  period  decrease  by  10%.  Quality  is  defined  as  an  increase  in 
compliance  with  customer  requirements  and  an  increase  of  on-time  delivery  (less  than 
10%  late  delivery). 


Quality,  as  defined  in  this  subgoal,  has  two  main  components: 

1 .  compliance  with  customer  requirements 

2.  on-time  delivery  of  customer  requirements 

To  illustrate  the  methodology,  the  following  section  presents  questions  and  indicators  for  the 
compliance  with  customer  requirements  component  of  the  quality  of  delivered  products 
subgoal  for  the  customer  quadrant  of  the  balanced  scorecard. 


Quality  Component  1 :  Compliance  with  Customer  Requirements 

The  following  questions  pertain  to  compliance  with  customer  requirements: 

1.  What  is  our  level  of  compliance  with  implementing  customer  requirements? 

2.  What  percentage  of  systems  is  in  full  compliance  with  stakeholder  requirements  (needs 
satisfied  equated  to  requirements  implemented). 

3.  Is  there  a  difference  in  compliance  with  customer  requirements  for  small,  medium,  and 
large  systems  or  projects? 

4.  Are  all  the  functional  requirements  desired  by  our  customers  being  delivered? 

5.  Is  the  level  of  compliance  a  function  of  the  size  of  the  project?  (Is  there  a  difference 
among  small,  medium,  and  large  projects?) 

6.  What  is  the  level  of  customer  satisfaction  for  completed  projects? 

Indicator  1:  Compliance  with  Requirements 

This  indicator  addresses  question  #1:  What  is  our  level  of  compliance  with  implementing 

customer  requirements? 
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Compliance  with  Requirements 
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Full  compliance  with  requirements  (100-95%) 
94  -  80%  compliance  with  requirements 
Less  than  80%  compliant  with  requirements 


The  proposed  indicator  shows  the  level  of  compliance  of  implementing  customer-specified 
requirements  for  completed  projects  per  reporting  period.  Completed  projects  in  a  specific 
time  period  are  rated  using  the  management-specified  assessment  values  as  shown  in  the  box 
below  the  indicator.  Full  compliance  of  customer  requirements  is  defined  as  100-95%.  For 
each  reporting  period,  the  numbers  in  each  box  indicate  the  number  of  completed  projects 
that  fall  into  each  grouping.  For  example,  in  period  1, 10  projects  were  in  full  compliance 
with  the  requirements.  This  represents  36%  of  the  projects  completed  during  that  reporting 
period. 

Since  multiple  periods  can  be  presented  on  the  chart,  a  trend  can  be  determined  if  the 
organization  is  improving  its  compliance  with  customer  requirements.  If  the  number  of 
completed  projects  per  reporting  period  is  low,  this  indicator  can  be  easily  modified  to  show 
the  number  of  completed  projects  per  year. 


Indicator  2:  Degree  of  Compliance  of  Customer  Requirements 

This  indicator  addresses  question  #2:  What  percentage  of  systems  is  in  full  compliance  with 
stakeholder  requirements? 
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This  display  presents  an  alternative  way  to  present  a  breakout  of  the  number  of  functional 
requirements  delivered  to  the  customer  compared  to  the  number  of  requirements  promised. 
Since  it  only  presents  a  snapshot  in  time,  no  trend  can  be  determined. 


Indicator  3:  Functional  Requirements  by  Product  Size 

This  indicator  addresses  question  #3:  Is  there  a  difference  in  compliance  with  customer 
requirements  for  small,  medium,  and  large  systems  or  projects? 
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This  indicator  is  a  more  detailed  breakout  of  the  information  in  the  previous  indicator.  Not 
only  does  this  indicator  break  out  the  level  of  compliance  with  customer  requirements  for 
small,  medium,  and  large  system,  it  provides  an  easy  overview  of  management-specified 
assessments  on  how  the  organization  is  doing  in  this  dimension.  If  the  compliance  with 
customer  requirements  for  the  system  is  100-95%,  it  is  rated  as  green;  94-85%  compliant 
with  requirements  is  rated  yellow,  and  less  than  84%  compliant  with  requirement  is  rated  red. 


Quality  Component  2:  On-Time  Delivery  of  Customer  Requirements 

The  following  questions  pertain  to  the  on-time  delivery  component  of  the  quality  of  customer 
requirements  subgoal: 

1.  Are  we  improving  our  delivery  times  for  small,  medium,  and  large  projects? 

2.  What  has  been  the  trend  in  delivery  time? 

3.  How  far  are  our  schedule  plans  from  actual  delivery  dates? 

4.  How  many  completed  projects  exceed  the  original  delivery  schedule  by  more  than  10%? 

5.  Is  there  a  difference  meeting  delivery  times  in  small,  medium,  and  large  projects? 
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Indicator  1:  Timeliness  of  Delivery  of  Customer  requirements 

This  indicators  address  question  #1:  Are  we  improving  our  delivery  times  for  small,  medium, 
and  large  projects? 


Deviation  in  Delivery  Times 
(Planned  Vs  Actual) 

%  Deviation 


Reporting  Periods 


This  display  provides  an  overview  on  the  trend  of  the  delivery  time  for  completed  projects 
segregated  into  three  size  categories — small,  medium,  or  large.  These  size  categorizations  are 
specified  by  the  organization  guidelines.  For  each  delivered  product,  the  difference  between 
the  schedule  and  actual  delivery  date  is  calculated  and  placed  into  the  appropriate  size 
category.  For  each  size  category,  the  average  deviation  in  delivery  time  is  calculated  and 
plotted. 

A  management  assessment  of  red,  yellow,  and  green  can  also  be  coded  into  the  display.  As 
shown  above,  deviation  of  delivery  times  for  large  projects  has  improved  by  going  from  red 
to  yellow  while  both  the  medium  and  small  projects  have  remained  relatively  constant. 


3.4.2  Subgoal:  Timeliness 
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Subgoal:  Timeliness.  Reduce  the  time  needed  to  transition  from  concept  to  operational 
capability  by  10%  in  the  next  delivery  (reduce  cycle  time). 


Questions: 

1.  What  have  been  the  delivery  times  of  completed  project  for  this  year? 

2.  What  has  been  the  trend  in  delivery  time  the  last  couple  of  years?  Are  we  improving? 

3.  Are  projects  being  delivered  on  time? 

4.  Has  there  been  an  improvement  in  meeting  delivery  dates? 

5.  What  is  the  delivery  time  trend  (cycle  time  trend)  for  small,  medium,  and  large  projects? 
Are  the  trends  the  same  for  the  different  project  effort  categories? 

6.  What  is  our  current  history  in  delivering  projects  on  time? 

The  following  section  provides  some  examples  of  indicators  that  address  some  of  these 
questions. 


Indicator  1:  Delivery  Time  Variance 

This  indicators  address  question  #1:  What  have  been  the  delivery  times  of  completed  project 
for  this  year? 


Delivery  Date  Variance 
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This  display  provides  an  overview  of  the  variance  between  planned  and  actual  delivery  times 
for  completed  projects.  As  each  project  is  completed,  the  variance  in  delivery  time  is 
calculated.  The  project  is  then  added  to  the  appropriate  variance  grouping.  Periodically  the 
number  of  projects  in  each  variance  grouping  is  summed  and  plotted  on  the  chart.  A  green, 
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yellow,  or  red  assessment  for  delivery  time  variance  can  be  defined  by  the  organization  and 
indicated  as  shown  on  the  display. 


Indicator  2:  Delivery  Time  Trends 

This  indicators  address  question  #5:  What  is  the  delivery  time  trend  (cycle  time  trends)  for 
small,  medium,  and  large  projects? 


Deviation  in  Delivery  Times 
(Planned  Vs  Actual) 

%  Deviation 
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Reporting  Periods 


The  same  indicator  for  addressing  the  timeliness  of  delivery  of  customer  requirements  can  be 
used  here.  This  is  an  example  of  how  one  indictor  can  answer  multiple  questions. 


3.4.3  Subgoal:  Resources 
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Subgoal:  Resources.  Increase  availability  and  capability  of  internal  resources  to  ensure 
customer  needs  are  satisfied  90%  of  the  time. 
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This  subgoal  is  composed  of  two  components: 

1.  availability  and  capability  of  internal  resources  (staff,  hardware,  software,  tools,  etc.) 

2.  customer  needs  satisfaction  (In  this  report,  needs  satisfaction  are  treated  as  requirements 
implemented.)  This  component  of  the  subgoal  was  addressed  earlier. 

Resources  Component  1:  Availability  and  Capability  of  Internal  Resources 

Questions  (availability  and  capability  of  internal  resources): 

1.  What  are  the  current  levels  of  personnel  assigned?  Are  they  adequate? 

2.  Are  sufficient  experienced  personnel  available  to  accomplish  the  mission  (number,  skill 
mix,  types)? 

•  skills:  entry-level,  journeyman,  high  grade 

•  types:  engineering  &  science,  technical,  other  disciplines 

•  workforce:  contractor,  civilian,  military 

3.  What  is  the  skill  level  of  the  current  staff? 

4.  What  are  the  personnel  availability  trends? 

5.  Is  the  recruiting/hiring  activity  sufficient  to  overcome  shortfalls?  Are  we  attracting  and 
retaining  key  skills? 

6.  Is  the  experience  level  sufficient  to  accomplish  the  job?  Will  additional  training  be 
required? 

A  large  number  of  indicators  can  be  postulated  to  address  these  questions.  Some  typical  ones 
are  shown  below. 


Indicator  1:  Skill  Mix  Indicators 

This  indicator  addresses  questions  such  as:  “Are  sufficient  experienced  personnel  available  to 
accomplish  the  mission?”  and  “What  is  the  skill  level  of  the  current  work  force?” 
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In  this  indicator,  the  staff  has  been  divided  into  three  major  groupings  (engineers  &  scientists, 
technical,  and  other  disciplines).  Each  of  these  groups  was  also  divided  by  skill  level  (entry- 
level,  journeyman,  or  high  grade).  The  indicator  provides  a  summary  view  of  current  and 
desired  strength  level  as  well  as  recruitment  activity. 

The  following  indicator  displays  the  current  numbers  for  entry-level,  journeyman,  and  high- 
grade  personnel  as  compared  to  their  target  values  (engineering  &  science  40%,  tech  45%, 
other  15%)  and  trend  values. 
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This  indicator  addresses  question  such  as:  What  are  the  comparisons  of  current  staff  numbers 
to  the  numbers  that  are  authorized? 
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This  indicator  addresses  the  same  question  as  above  but  also  presents  trend  information,  such 
as  the  number  of  staff  for  fiscal  years  2001  through  2003. 


Indicator  2:  Current  Breakout  of  Workforce 

This  indicator  provides  a  snapshot  of  the  current  workforce  and  addresses  the  question: 
“What  is  the  staffing  breakout  of  the  current  workforce?”  This  type  of  information  could  also 
be  presented  in  table  form. 
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4  Summary 


The  balanced  scorecard  and  GQ(I)M  methodologies  provides  a  systematic  way  to  obtain 
measures  and  indicators  reflecting  the  health  and  performance  of  an  organization.  The 
approach  described  here  uses  an  organization’s  vision  and  mission  statements  to  identify  and 
clarify  strategic  goals  and  subgoals.  In  this  iterative  approach,  these  goals  and  subgoals  are 
mapped  to  the  balanced  scorecard  and  refined  as  necessary.  The  GQ(I)M  methodology  then  is 
used  to  identify  measures  and  indicators  for  each  dimension  of  the  scorecard. 

An  example  application  of  the  approach  was  discussed  for  a  “typical  organization.”  From 
this  example  application,  sample  indicators  for  the  customer  quadrant  of  the  balanced 
scorecard  were  derived  and  presented.  For  the  purpose  of  brevity,  measures  and  indicators  for 
the  other  quadrants  were  not  presented  here. 

Again  we  note  that  the  measures  and  indicators  resulting  from  the  example  application  are 
only  examples  to  illustrate  the  methodology  and  should  not  be  adopted  as-is  by  organizations. 

Since  the  number  of  potential  measures  and  indicators  may  be  large,  we  recommend  that  a 
minimum  set  be  used  as  a  starting  point.  The  approach  proposed  here  contains  numerous 
steps  that  allow  the  set  of  measures  and  indicators  to  be  prioritized  and  thereby  reduced  to  a 
minimum  number  consistent  with  an  organization’s  business  needs  and  goals.  We  believe  that 
such  a  recommended  set  still  provides  a  general  overview  of  the  issues  being  addressed  in 
each  quadrant  of  the  balanced  scorecard  to  accurately  portray  an  enterprise-wide  indication  of 
the  health  of  an  organization. 

There  are  two  additional  activities  to  be  considered.  First,  the  measures  need  to  be 
“normalized.”  For  example,  the  measure  “on-time  delivery  of  products”  in  our  typical 
organization  makes  sense  if  there  is  a  planned  delivery  schedule  to  compare  it  against.  Each 
of  the  measures  should  be  augmented  similarly.  In  addition,  the  measures  are  only  useful  if 
the  organization  institutes  a  program  or  methodology  to  collect  data  required  for  the 
measures. 

We  defer  discussions  on  techniques  to  analyze  resultant  indicators  and  to  develop  plans  to 
achieve  the  organization’s  business  goals  for  subsequent  technical  notes. 
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Appendix  A  GQ(I)M  Methodology 


Methodology 

The  goal-driven  measurement  methodology  as  implemented  at  the  SEI  consists  of  the 
following  steps: 


Identify  Goals 

1 .  identify  your  business  goals 

2.  identify  what  you  want  to  know  or  leam 

3.  identify  your  subgoals 

4.  identify  the  entities  and  attributes  related  to  your  subgoals 

5.  formalize  your  measurement  goals 

Define  Indicators 

1.  identify  quantifiable  questions  and  the  related  indicators  that  you  will  use  to  help  you 
achieve  your  measurement  goals 

2.  identify  the  data  elements  that  you  will  collect  to  construct  the  indicators  that  help 
answer  your  questions 

3.  define  the  measures  to  be  used,  and  make  these  definitions  operational 

Create  an  Action  Plan 

1.  identify  the  actions  that  you  will  take  to  implement  the  measures 

2.  prepare  a  plan  for  implementing  the  measures 

The  goal-driven  software  measurement  approach  is  described  in  the  SEI’s  Goal-Driven 
Software  Measurement  Guidebook  [Park  96]. 

Figure  5  depicts  the  general  steps  we  use  to  derive  the  indicators  and  measures. 
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Step  10:  Prepare  a  plan 


Figure  5:  Goal-Driven  (Indicator)  Measurement  Methodology 

The  following  is  a  brief  description  of  each  step. 


Step  1:  Identify  Business  Goals.  In  this  step,  the  business  goals  that  drive  your 
organization’s  effort  are  identified.  Without  a  clear  sense  of  the  organization’s  strategic  goals 
and  the  objectives  and  responsibilities  for  each  work  unit  or  position,  there  is  a  risk  that 
measures  will  not  be  aligned  with  important  issues  in  your  organization. 


Step  2:  Identify  What  You  Want  to  Know  or  Learn.  If  measurement  activities  are  to  be 
aligned  with  your  goals,  then  we  must  translate  the  goals  into  operational  statements.  In  this 
step  the  goals  are  linked  with  knowledge  of  the  organizations  business  strategies  and 
processes. 

Step  3:  Identify  Your  Subgoals.  The  preceding  step  usually  generates  many  questions.  By 
analyzing  the  questions  and  seeking  commonality  among  them,  subgoals  can  be  derived.  The 
subgoals  provide  a  refinement  of  the  goal  and  serve  as  a  summary  for  the  questions  to  which 
you  would  like  answers. 

Step  4:  Identify  the  Entities  and  Attributes.  The  subgoals  and  related  questions  define  the 
focus  for  the  measures.  Analysis  of  the  questions  will  identify  what  needs  to  be  measured. 


Step  5:  Formalize  Your  Measurement  Goals.  In  this  step,  a  measurement  goal  is  crafted 
that  merges  the  purpose  and  perspective  derived  from  the  business  goal  with  the  possibilities 
for  measurement  as  they  exist  within  the  organization’s  work  processes.  In  addition,  the  goal 
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statements  express  environmental  or  contextual  factors  that  are  important  for  those  who  will 
design  and  do  the  measurement  and  analysis  activities. 

Step  6:  Identify  Quantifiable  Questions  and  the  Related  Indicators.  In  this  step, 
indicators  or  displays  to  address  the  goal  are  sketched  out.  Sketching  or  drafting  the  table, 
chart,  or  report  that  needs  to  be  produced  helps  to  make  sure  the  requirements  for 
measurement  are  complete. 

Step  7:  Identify  the  Data  Elements.  In  this  step,  the  data  elements  required  to  construct  the 
indicators  are  identified.  The  existence  and  sources  of  the  needed  data  are  assessed. 

Step  8:  Define  the  Measures.  Definitions  are  critical  for  achieving  proper  interpretations  of 
the  data.  The  definitions  need  to  be  created  with  the  purpose  of  the  indicator  in  mind  and 
should  consistently  provide  an  answer  to  the  question  that  the  indicator  addresses. 
Developing  a  complete  and  unambiguous  (minimally  ambiguous)  definition  can  be  arduous. 
To  aid  this  task,  the  SEI  developed  a  series  of  measurement  framework  checklists  for 
common  software  measures  such  as  size,  effort,  milestones,  and  defects  [Park  et  al.  92], 
[Goethert  et  al.  92],  [Florae  92]. 

Step  9:  Identify  the  Actions  for  Implementation.  Knowing  the  data  needed  and  having 
defined  them,  the  existing  situation  within  the  organization  is  analyzed  with  respect  to  the 
measurement  needs. 

Step  10:  Prepare  an  Action  Plan.  Once  a  gap  analysis  has  been  completed  between  the 
needed  data  and  the  existing  measurement  activities,  an  action  plan  can  be  prepared. 
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Appendix  B  Mission,  Vision,  and  Goal  Statements4 


Mission  Statements 

Although  not  all  authors  may  agree  on  the  best  form  for  an  effective  mission  statement,  most 
believe  that  it  should  define  at  least  the  following  three  items: 

1.  The  stakeholders  of  the  group’s  work.  This  is  defined  not  in  terms  of  some  segment  of 
the  organization  per  se,  but  in  terms  of  a  basic  defining  need  premise  that  leads  that 
person  (or  entity)  to  consider  using  the  products  or  services  of  your  group. 

2.  The  value  premise.  Define  this  not  in  terms  of  what  your  group  does  or  produces,  but  in 
terms  of  the  fundamental  value  it  represents  in  matching  the  stakeholders’  need  premise. 

3.  What  makes  you  special.  This  is  your  special  means  for  creating  value  to  have  your 
stakeholders  want  to  continue  working  with  you  and  using  your  products  and  services. 

After  you  draft  the  mission  statement,  it  makes  sense  to  evaluate  it  carefully  using  these 
criteria: 

•  definitive.  It  defines  the  stakeholders  and  their  need  premise,  the  value-delivery  premise 
to  be  offered,  and  the  means  for  putting  the  two  together.  It  tells  the  story  of  your  group’s 
way  of  doing  business. 

•  concise.  It  makes  the  point  in  one  fairly  simple  paragraph.  A  basic  mission  statement 
should  be  something  you  can  easily  write  on  the  back  of  a  business  card. 

•  actionable.  It  gives  a  person  reading  it  some  idea  of  what  it  looks  like  in  operation,  and 
what  kinds  of  actions  are  involved  in  delivering  it. 

Vision  Statements 

Creating  a  meaningful  vision  statement  seems  to  be  one  of  the  most  challenging  and 
frustrating  tasks  that  a  group  faces  when  coming  together  as  a  team.  But  it’s  a  mistake  to  just 
push  on  without  a  clear  statement  of  vision. 

The  description  of  that  end-state  is  called  a  vision  statement.  The  vision  may  be  revised  as 
times  goes  on.  A  vision  is  an  image  of  what  the  people  in  the  group  aspire  for  it  to  become. 


4  This  appendix  is  drawn  from  a  white  paper  authored  by  Mark  Kasunic  titled  “Vision,  Mission, 
Goals,  Objectives,  Activities”  and  a  paper  by  Tom  Keenze,  dated  October  17, 2000,  available  at 
http://staff.ed.uiuc.edu/rthomas/oetmission/statement.html. 
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Note  that  the  key  word  is  image.  It  must  be  something  that  you  can  describe  and  that  people 
can  see  in  their  mind’s  eye. 

If  the  group  can’t  come  to  consensus  on  a  statement  that  expresses  the  meaning  of  their 
group,  then  what  can  they  hope  to  say  to  the  rest  of  the  organization  that  will  make  any 
sense?  If  all  they  have  are  platitudes,  then  they  have  no  message.  In  many  ways,  the  vision 
statement  is  a  test  of  whether  there  really  is  a  message. 

Although  the  group  may  have  a  pretty  good  idea  of  what  they  want  to  do  to  improve  the 
organization,  it  is  easy  to  lose  sight  of  this  when  dealing  with  the  day-to-day  hassles  that 
plague  all  organizations.  The  vision  statement  keeps  the  endpoint  in  sight.  It  also  lets  others 
know  what  the  group  stands  for  and  what  it  wants  to  do.  It  helps  to  keep  group  members 
focused  and  bound  together  in  common  purpose. 

Visioning  begins  with  the  future,  not  the  present.  It  focuses  on  the  end-state,  not  the  means  of 
getting  there.  It  is  a  mental  picture  of  the  group,  operating  in  an  environment,  performing  to 
some  criterion  of  excellence,  and  appreciated  for  what  it  contributes.  A  vision  is  not  a  plan.  A 
vision  is  knowing  what  you  want  to  do.  The  “how  to”  comes  later. 

What  makes  an  effective  vision  statement?  Three  components  help  to  make  a  vision 
statement  valid  and  useful  for  people: 

1.  A  focused  concept — something  beyond  platitudes;  a  value  creation  premise  that  people 
can  actually  picture  as  existing. 

2.  A  sense  of  noble  purpose— something  that  is  really  worth  doing;  something  that  can 
create  value,  make  a  contribution,  make  the  world  a  better  place  in  some  way,  and  win 
people’s  commitment. 

3.  A  plausible  chance  of  success — something  people  can  realistically  believe  to  be 
possible  and,  if  not  perfectly  attainable,  at  least  plausible  to  strive  for. 

Decide  what  you’re  not  about  and  make  sure  your  vision  statement  excludes  it.  A  common 
mistake  is  to  make  the  vision  too  broad  just  in  case  you  want  to  expand  it  later.  It  is  better  to 
limit  the  vision  and  provide  needed  direction.  Radical  departures  from  the  original  vision  can 
then  be  challenged  effectively  (which  is  the  whole  point),  and  the  vision  formally  expanded 
to  include  them  if  necessary. 

A  quick  hint  for  developing  vision  and  mission  statements:  The  most  common  cause  of 
the  frustrating  wrangles  that  group  members  get  into  is  confusing  editing  with  thinking. 

When  starting  to  draft  a  vision  or  mission  statement  directly,  teams  frequently  fall  into 
debates  about  the  best  choice  of  words  around  the  idea.  Separate  the  thinking  from  the 
“wordsmithing.”  The  better  way  is  to  work  out  the  basic  idea  of  the  vision  and  mission 
statements  first,  and  then  have  somebody  put  some  compelling  words  around  the  idea.  You’re 
not  ready  to  start  drafting  the  language  of  your  vision  or  mission  statement  until  you  have 
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settled  on  a  driving  idea,  the  organizing  principle  behind  the  way  you  want  to  do  your  work, 
and  until  you  can  write  that  idea  down  on  the  back  of  a  business  card.  Once  you  have 
consensus  on  the  core  concept  of  your  group,  writing  it  out  becomes  a  much  more 
manageable  process.  There  is  always  a  sudden  rise  in  energy  and  enthusiasm  among  group 
members  at  the  point  when  they  break  through  to  the  critical  premise  of  the  group.  After  that, 
they’re  much  more  willing  to  push  on  with  the  process  of  getting  the  language  right. 


Goals 

Goals  typically  represent  an  end  state  similar  to  the  vision.  However,  this  is  not  always  true. 
There  may  be  goals  that  represent  an  ongoing  state  such  as  “improve  the  communication 
internal  to  the  organization.”  However,  goals  should  be  achievable,  which  implies  that  there 
needs  to  be  some  way  to  realize  or  measure  that  the  goals  have  been  achieved.  This  means 
that  the  statement  of  goals  should  contain  a  qualitative  or  quantitative  portion  that  allows 
their  achievement  to  be  measured  in  some  meaningful  way. 
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Appendix  C  Indicators 


Indicators 

An  indicator  is  defined  as  a  measure  or  combination  of  measures  that  provides  insight  into  a 
process,  a  project,  or  a  product.  An  indicator  is  usually  a  graph  or  table  that  you  define  for  the 
organization’s  needs.  Indicators  frequently  make  a  comparison  between  two  values,  such  as 
planned  and  actual  values. 


Classification  of  Indicators 

There  are  three  main  classifications  of  indicators.  These  are  defined  as  follows: 

1.  Success  Indicators:  These  indicators  are  constructed  from  the  defined  success  criteria 
and  are  used  to  determine  if  the  goals  have  been  met. 

2.  Progress  Indicators:  These  indicators  are  used  to  track  the  progress  or  execution  of  the 
defined  tasks.  A  Gantt  chart  is  a  good  example.  The  successful  execution  of  all  the 
defined  tasks  does  not  necessarily  guarantee  that  the  goal  has  been  successfully  met. 

3.  Analysis  Indicators:  These  indicators  are  used  to  assist  in  analyzing  the  output  of  each 
task.  The  analyses  help  test  our  assumptions  about  the  data  we  are  using  to  judge 
progress  and  success. 


The  following  figure  illustrates  the  difference  among  types  of  indicators: 
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Figure  6:  Classification  of  Indicators 
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Many  organizations  have  difficulty  deciding  how  to  tell  if  or  when  their  business  goals  have 
been  achieved.  While  organizations  are  often  able  to  articulate  a  strategy  and  define  tasks  for 
achieving  their  goals,  they  have  difficulty  understanding  the  difference  between  success 
indicators  (indicators  used  to  determine  if  the  goals  have  been  met)  and  progress  indicators 
(indicators  used  for  tracking  the  execution  of  tasks).  Organizations  often  use  these  indicators 
to  track  the  execution  of  tasks  and  as  a  proxy  for  measuring  if  a  goal  has  been  achieved. 
When  all  the  tasks  have  been  executed,  organizations  declare  success — their  goals  have  been 
met.  But  they  often  neglect  to  analyze  the  outcome  of  the  tasks  to  determine  if  their  goals 
have  been  met  successfully.  Execution  of  the  defined  tasks  is  a  necessary  but  not  sufficient 
condition  for  meeting  the  goal. 


Guidance  on  Generating  Indicators 

The  following  guidance  should  be  kept  in  mind  for  generating  useful  indicators  [PSM  00]. 

•  Use  consistent  conventions 

•  Keep  it  simple:  keep  the  message  clear 

•  Unique  titles  should  reflect  insight  and  scope 

•  Include  AS  OF  line  or  date 

•  Label  each  axis  and  provide  scale  markers 

•  Annotate  with  milestones  and  significant  events 

•  Use  same  axes  and  scales  if  indicators  will  be  compared 
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Appendix  D  Indicator  Template 


Indicator  Template 

Using  the  information  identified  during  the  workshop,  a  strawman  indicator  template  for  each 
identified  indicator  can  be  completed.  The  template  addresses  and  documents  the  why,  what, 
who,  where,  how,  etc.  for  each  indicator.  Each  indicator  should  be  documented  using  the 
indicator  template  shown  in  the  figure  below. 


INDICATOR  TEMPLATE 
Measurement  Goal#:  — — — - 


Objective 
Questions 
Visual  Display 


Perspective 

!nput(s) 


Data  Elements 


Responsibility 
for  Reporting 
Form($) 
Algorithm  - 


Assumptions  — - 

Interpretation 

X-reference 
Probing  Questions 
Evolution 


Figure  7:  Indicator  Template 


A  description  of  each  field  in  the  indicator  template  follows. 
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INDICATOR  TEMPLATE 


Indicator  Name/Title: 


Date 


OBJECTIVE  Describe  the  objective  or  purpose  of  the  indicator. 

QUESTIONS  List  the  question(s)  the  indicator  user  is  trying  to  answer.  Examples: 

Is  the  project  on  schedule?  Is  the  product  ready  to  ship?  Should  we 
invest  in  moving  more  software  organizations  to  CMM  maturity 
level  3? 


VISUAL  DISPLAY  Provide  a  graphical  view  of  the  indicator. 


PERSPECTIVE  Identify  the  target  audience  of  this  indicator.  (Who  is  going  to  use 
this  information?) 

INPUT(S) 

Data  Elements  List  the  data  elements  that  are  used  in  the  production  of  the  indicator. 

The  description  should  be  consistent  to  the  degree  that  different 
individuals  in  different  environments  can  reliably  generate  comparable 
numbers  in  the  same  units.  In  many  cases,  organizations  will  want  to 
collect  more  detailed  data  (e.g.,  additional  attributes  of  trouble  reports) 
and  the  inputs  should  be  described  in  such  a  way  as  to  permit  these 
additional  attributes  as  well  as  the  “common”  attributes. 


Total 
Attributes 
for  a  Data 
Element 


Common 

Site  Specific 

Attributes 

Attributes 

-  Attr.  1 

-  Attr.  1 

-  Attr.  2 

+ 

-  Attr.  2 

• 

• 

• 

• 

• 

• 

-  Attr.  N 

-  Attr.  M 
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Responsibility 

for  Reporting  Indicate  who  has  responsibility  for  reporting  the  data. 

Forms  Reference  any  standard  forms  for  data  collection  (if  applicable)  and 

provide  information  about  where  to  obtain  them. 

Specify  the  algorithm  or  formula  required  to  combine  data 
elements  to  create  input  values  for  the  display.  It  may  be 
very  simple,  such  as  Input  1/Input2,  or  it  may  be  much  more 
complex.  Include  how  the  data  is  plotted  on  the  graph. 

Identify  any  assumptions  about  the  organization,  its 
processes,  life-cycle  models,  and  so  on  that  are  important 
conditions  for  collecting  and  using  this  indicator. 

INTERPRETATION  Describe  what  different  values  of  the  indicator  mean.  Make 

it  clear  how  the  indicator  answers  the  “Questions”  section 
above.  Provide  any  important  cautions  about  how  the  data 
could  be  misinterpreted  and  measures  to  take  to  avoid 
misinterpretation. 

X-reference  If  the  values  of  other  defined  indicators  influence  the 

appropriate  interpretation  of  the  current  indicator,  refer  to 
them  here. 

Probing  Questions  List  questions  that  delve  into  the  possible  reasons  for  the 

value  of  an  indicator,  whether  performance  is  meeting 
expectations  or  whether  appropriate  action  is  being  taken. 

Evolution  Specify  how  the  indicator  can  be  improved  over  time, 

especially  as  more  historical  data  accumulates  e.g.,  by 
comparison  of  projects  using  new  processes,  tools, 
environments  with  a  baseline;  using  baseline  data  to 
establish  control  limits  around  some  anticipated  value  based 
on  project  characteristics. 


ALGORITHM 


ASSUMPTIONS 
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Modified  Indicator  Template 

A  modified  template  for  describing  the  indicators  is  shown  below.  This  indicator  template 
illustrates  how  an  indicator  template  may  be  tailored  to  add  other  specific  data  requirements. 
New  fields  are  shown  in  blue. 


MODIFIED  INDICATOR  TEMPLATE 


Date 


Indicator  Name/Title: 


OBJECTIVE  Describe  the  objective  or  purpose  of  the  indicator. 

QUESTIONS  List  the  question(s)  the  indicator  user  is  trying  to  answer.  Examples: 

Is  the  project  on  schedule?  Is  the  product  ready  to  ship?  Should  we 
invest  in  moving  more  software  organizations  to  CMM  maturity 
level  3? 


VISUAL  DISPLAY  Provide  a  graphical  view  of  the  indicator. 


PERSPECTIVE  Identify  the  target  audience  of  this  indicator.  (Who  is  going  to 
use  this  information?) 

INPUTS 


Data  Elements 


Definition 


List  all  the  data  elements  in  the 
production  of  the  indicator. 


Precisely  define  the  data  element  used  or 
point  to  where  the  definition  can  be  found. 


DATA  COLLECTON 
How 

When/How  Often 
By  Whom 


Describe  how  the  data  will  be  collected. 

Describe  when  the  data  will  be  collected  and  how  often. 
Specify  who  will  collect  the  data  (an  individual,  office,  etc.) 
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Forms 

DATA  REPORTING 

Responsibility 
for  Reporting 

By/To  Whom 
How  Often 

ALGORITHM 

ASSUMPTION 

ANALYSIS 

INTERPRETATION 

PROBING  QUESTIONS 

EVOLUTION 


Reference  any  standard  forms  for  data  collection  (if 
applicable)  and  provide  information  about  where  to  obtain 
them. 


Indicate  who  has  responsibility  for  reporting  the  data. 

Indicate  who  will  do  the  reporting  and  to  whom  the  report  is 
going.  This  may  be  an  individual  or  an  organizational  entity. 

Specify  how  often  the  data  will  be  reported  (daily,  weekly, 
monthly,  as  required,  etc.) 


Specify  the  algorithm  or  formula  required  to  combine  data 
elements  to  create  input  values  for  the  display.  It  may  be 
very  simple,  such  as  Input  1/Input2,  or  it  may  be  much  more 
complex.  It  should  also  include  how  the  data  is  plotted  on 
the  graph. 

Identify  any  assumptions  about  the  organization,  its 
processes,  life-cycle  models,  and  so  on  that  are  important 
conditions  for  collecting  and  using  this  indicator. 


Specify  what  type  of  analysis  can  be  done  with  the 
information. 

Describe  what  different  values  of  the  indicator  mean.  Make 
it  clear  how  the  indicator  answers  the  “Questions”  section 
above.  Provide  any  important  cautions  about  how  the  data 
could  be  misinterpreted  and  measures  to  take  to  avoid 
misinterpretation. 

List  questions  that  delve  into  the  possible  reasons  for  the 
value  of  an  indicator,  whether  performance  is  meeting 
expectations  or  whether  appropriate  action  is  being  taken. 

Specify  how  the  indicator  can  be  improved  over  time, 
especially  as  more  historical  data  accumulates  e.g.,  by 
comparison  of  projects  using  new  processes,  tools, 
environments  with  a  baseline;  using  baseline  data  to 
establish  control  limits  around  some  anticipated  value  based 
on  project  characteristics. 
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X-REFERENCES 


If  the  values  of  other  defined  indicators  influence  the 
appropriate  interpretation  of  the  current  indicator,  refer  to 
them  here. 
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Appendix  E  Glossary 


goal  the  objective  to  which  an  endeavor  is  directed 

indicator  a  measure  or  combination  of  measures  that  provides  insight  into  a  process,  a 
project  or  a  product  itself.  An  indicator  is  usually  a  graph  or  table  that  you 
define  for  your  program  needs.  An  indicator  is  a  measure  or  a  group  of 
measures  the  provide  information  about  a  project  issue.  Indicators  frequently 
make  comparison  between  two  values,  such  as  planned  and  actual  values. 

indicator  examples: 

Example  of  Risk  Impact 


j>  1  Moderate 


1  2  3  4  5 

Consequence  of  Failure 


Earned  Value  Management 


Target  Area 


Behind  Schedule  end  Overspent 


2 

0.82  0.86  0.90  0.94  0.98 


Ahead  of  Schedule  end  Overspent 
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measurement  a  standard  or  unit  of  measurement 

a  quantitative  indication  of  the  extent,  amount,  dimension,  capacity  or  size  of 
some  attribute  of  a  product  or  process 

measurement  examples: 

•  CMM-Based  Assessment  for  Internal  Process  Improvement  (CBA-IPI) 

•  function  point  counting 

•  time  reporting 

measure  the  process  by  which  numbers  or  symbols  are  assigned  to  attributes  of 

entities  in  the  real  world  in  such  a  way  as  to  characterize  the  attributes  by 
clearly  defined  rules5 

measure  examples: 

•  number  of  defects 

•  source  lines  of  code  (SLOC) 

•  person  hours 

•  dollars 

•  number  of  requirements 


metric  a  quantitative  measure  of  the  degree  to  which  a  system,  component,  or 

process  possess  a  given  attribute6 

metric  examples: 

•  defects  per  1,000  SLOC 

•  number  of  change  requests  per  phase 

•  defects  per  phase 


5  This  is  adapted  from  Fenton,  Norman.  Software  Metrics:  A  Rigorous  Approach.  London:  Chapman 
and  Hall,  1991. 

6  This  is  adapted  from  the  I-EEE  Standard  Glossary  of  Software  Engineering  Terminology. 
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Appendix  F 

[Basili  88] 

[Goethert  01] 

[McGarry  93] 

[Rombach  89] 

[Solingen  99] 

[Springsteen  94] 


Recommended  Reading 


Basili,  Victor  R.  &  Rombach,  Dieter  H.  ‘The  TAME  Project:  Towards 
Improvement-Oriented  Software  Environments.”  IEEE  Transactions  on 
Software  Engineering  14,  6  (June  1988):  758-773. 

Goethert,  Wolfhart  &  Hayes,  Will.  Experiences  in  Implementing 
Measurement  Programs  (CMU/SEI-2001-TN-026,  ADA3399165). 
Pittsburgh,  PA:  Software  Engineering  Institute,  Carnegie  Mellon 
University,  2001.  <http://www.sei.cmu.edu/publications/documents 
/0 1  .reports/0 1  tn026.html> 

McGarry,  John.  Software  Development  Metrics  Qualitative  Assessment 
Structure.  Newport,  RI:  Naval  Undersea  Warfare  Center,  1993. 

Rombach,  Dieter  H.  &  Ulery,  Bradford  T.  “Improving  Software 
Maintenance  Through  Measurement.”  Proceedings  of  the  IEEE  77, 4 
(April  1989):  581-595.  <http://ieeexplore.ieee.org/xpl 
/tocresult.jsp?isNumber=91 1> 

Solingen,  Rini  Van  &  Berghout,  Egon.  The  Goal/Question/Metric 
Method.  London:  McGraw-Hill  Publishing  Company,  1999. 

Springsteen,  Beth  et  al.  “Survey  of  Software  Metrics  in  the  Department 
of  Defense  and  Industry,”  Institute  for  Defense  Analyses,  IDA  Paper  P- 
2996,  April  1994. 
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